This repository was archived by the owner on May 9, 2026. It is now read-only.
feat: ingest synced skill documents into knowledge graph#9
Merged
Merged
Conversation
…tion - Added `ingestNewEmails` function in Gmail sync to process unsubmitted emails and integrate them into the knowledge graph. - Introduced `ingestNewDocuments` function in Notion sync for ingesting unsubmitted pages and database rows into the knowledge graph. - Enhanced both functions to handle sensitive emails and ensure proper metadata is included during ingestion. - Updated sync processes to call these new ingestion functions after completing email and document synchronization.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
✅ Files skipped from review due to trivial changes (2)
📝 WalkthroughWalkthroughAdded direct ingestion flows to Gmail and Notion syncs that import newly synced items into the knowledge graph; also updated live-test import paths/usages, adjusted server-ping tests to use localhost, awaited ping-now’s doPing, and advanced the openhuman submodule pointer. Changes
Sequence Diagram(s)sequenceDiagram
participant Sync as Sync Module
participant DB as Database
participant Memory as Knowledge Graph
Sync->>DB: getUnsubmittedItems(INGEST_QUERY_LIMIT)
DB-->>Sync: return items
Sync->>DB: markSensitiveAsSubmitted(ids) (Gmail)
DB-->>Sync: ack
loop per item
Sync->>Sync: extract content (body_text || snippet / content_text)
alt content exists
Sync->>Memory: memory.insert(title, content, metadata, timestamps)
Memory-->>Sync: inserted
else no content
Sync-->>Sync: mark as submitted (skip insert)
end
Sync->>Sync: collect processed ID
end
Sync->>DB: markItemsSubmitted(all_processed_ids)
DB-->>Sync: ack
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
- Move gmail/notion live integration scripts out of __tests__/ so the test runner no longer picks them up (they require JWT_TOKEN) - Fix server-ping setup test: use localhost:7799 (the running test runtime) instead of unreachable fake domains - Fix ping-now tool: await doPing() so the DB query runs after the ping is logged Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
senamakel
force-pushed
the
feat/skill-sync-ingestion
branch
from
5f0804b to
62f6ab2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Skill Submission
Skill name:
notion,gmailType: [x] Coded (skill.ts)
Description
Wire Notion and Gmail skill sync flows into the memory ingestion pipeline so that synced documents are automatically processed into the knowledge graph via
memory.insert().Problem: Skills sync documents into their local SQLite DB but the content never reaches the Rust ingestion pipeline (GLiNER entity/relation extraction → knowledge graph). The Notion skill had a
backend.submitData()call that was commented out, and Gmail had no submission flow at all — despite both having all the DB tracking infrastructure (backend_submittedflag,getUnsubmitted*,markSubmittedhelpers) already in place.Solution: Replace the old
backend.submitData()batching approach with per-documentmemory.insert()calls, which route through the existing Rust pipeline:put_doc()→IngestionQueue→ GLiNER extraction → graph storage.Notion (
src/core/notion/sync.ts)submitNewData()(commented-outbackend.submitData+ chunking/batching) withingestNewDocuments()memory.insert()per unsubmitted page and database rowdocumentId: "notion-page-{id}"/"notion-row-{id}"for dedup/upsertsourceType: 'doc'with page metadata (url, timestamps, parent info)backend_submittedtrackingGmail (
src/core/gmail/sync.ts)ingestNewEmails()functionmemory.insert()per unsubmitted email withsourceType: 'email'documentId: "gmail-email-{id}"for dedupperformInitialSync()andonSync()(incremental) flowsChecklist
eval(),Function(), or dynamic code executionyarn typecheckpassesyarn buildpassesTesting
yarn typecheckpasses with no errorsyarn buildcompletes successfully (all 3 skills bundle)memory.insert()bridge is fire-and-forget async on the Rust side — no blocking risk during syncCategory
🤖 Generated with Claude Code
Summary by CodeRabbit